The RAG Cost Curve: When an index beats live search

Every retrieval system now competes with a simpler-looking alternative: skip the index and let the agent search live over tools with long context. For small or fast-changing corpora that genuinely makes sense. But “just search live” has a cost curve with a very different shape from an index, and most teams pick a side before doing the maths.

This talk makes the maths explicit and reproducible. I’ll walk through recall, latency and memory tradeoffs measured across quantization schemes and dimensionality reduction: how far you can compress an index without trashing recall, and where the refinement step is quietly doing the real work. Then the part most benchmarks skip: where recall stops predicting end-task answer quality. Recall is a proxy, and I’ll show where it breaks.

From there I convert compression into money, against repeated live search and long-context inference over the same corpus. Benchmarks run on public datasets; the methodology and numbers are open.

I’ll ground it in two open-source systems built on exactly this tradeoff: memsearch, an agent-memory layer that indexes plain markdown instead of replaying full transcripts, and claude-context, a semantic code-search tool that avoids loading whole repositories into a model’s context on every query. Both exist because live re-search over large private history or repos gets expensive fast, and both come with measured token and latency deltas.

You’ll leave with a practical framework for the question that actually matters: at what corpus size, update frequency and query volume a compressed index beats live search on cost, and how much answer quality you trade to get there.

Simon Hearne

Zilliz (Milvus)

Simon is a Solutions Architect at Zilliz, working on vector search architecture, migration and performance for production retrieval systems. Previously at Elastic and Akamai, with an MEng in AI and an MSc in Data Science. Simon has a background in web performance and observability which shapes how he thinks about search: balancing accuracy, latency and cost whilst measuring the outcome.